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Teaching Program Evaluation on Interactive Television 

Keith McNeil 
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Jim Steinhauser 

Region 19 Head Start Research Unit 
Isadore Newman 
University of Akron 

Abstract 

A five-session course entitled, “Program Evaluation” was taught via Interactive 
Television during the summer of 2002 to 68 Doctoral and Masters students in 6 locations 
throughout the state of New Mexico. Five of the locations received the instruction though 
Interactive Television. Students received a 4-hour lecture session and then participated in 
off-line activities directed by the instructor. Problems in distance education delivery and 
recommended solutions are provided. 

Introduction 

We would like to share with you our experience teaching a course on Program 
Evaluation using Interactive Television(ITV). First we will present the context of the course, 
then problems related to technology, and proposed solutions. 

Context of the course 

A new doctoral program in Community College Administration was initiated in 
January 2002. The program was designed to be delivered entirely in a distance education 
mode. The delivery of the program was promised to be a mix of ITV, WebCT, phone, e- 
mail, and travel by faculty to the students across the state of New Mexico. See Appendix 
A for the program description. 

NMSU faculty marketed the program during the summer of 2001 , perhaps a little 
too successfully. A total of 62 students were accepted into the program and began their 
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doctoral work in January of 2002. (In addition, 6 students were accepted into the masters 
program and agreed to take the same courses.) They came to the main campus of NMSU 
in Las Cruces for a two-day orientation, and the plan was for them to return in about three 
years to walk across the stage. The remainder of the courses were to be delivered by 
distance education to the five sites in Appendix B. 

Students take two courses every semester and three in the summer. Most of them 
were released from their duties for Friday afternoon. Courses were offered Friday afternoon 
and Saturday morning, each 4 hours long. Because there were 5 designated course times 
(5 * 4 = 20), instructors had to develop activities that supported instruction for the other 25 
hours that normally constitute a 3-hour course. These activities had to document student 
participation. 

Why us, Lord? 

We are not based in the Educational Administration department, but have a lot of 
ties to the department. All of their doctoral students take the 3 course statistics sequence, as 
well as the dissertation writing course from McNeil. He plays racquetball twice a week with 
the professor that originated the idea, Dr. Roy Rodriguez. When he indicated that they had 
a Program Evaluation course in the curriculum, he volunteered Steinhauser and himself to 
teach it for the following reasons: 

1 . He was in the late stages of writing a textbook on Program Evaluation (with 
Isadore Newman), and this would be an impetus to finish the draft. 

2. The course would give them a chance to try out the text with a live audience. 

3. The focus of the text is on the collaboration of the Program Administrator with the 
evaluator, a focus that likely would be of interest to those studying to be Community 
College Administrators. 

4. McNeil wanted to see what ITV was all about. 

5. McNeil and Steinhauser didn’t have much else to do in the summer of 2002. 

What were the facilities like? 

Instruction occurred in a room in the Business Building on the other side of campus. 
The room contained an overhead projector, computer for Power Point presentations, video 
equipment, 4 Televisions, and camera that would follow the instructor or switch to someone 
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talking in the room. (See Appendix C.) There were 4 students in the room, 3 driving from 
their residence 70 miles away. They could look at the instructor, or one of the TVs (Instructor 
TV in Appendix C) that contained the feed that all the other students across the state saw. 
The other TV (identified as Remote TV in Appendix C ) focused on the student (wherever 
in the state) who was currently speaking, or who had last spoken. 

The Assistant at the back of the room, either McNeil or Steinhauser depending on 
who was teaching at the time, had collaborated on program evaluations for the past 4 years. 

After McNeil attempted to teach for 4 straight hours for the first class session, Dr. 
Steinhauser was invited to share the teaching responsibilities, which he did admirably. The 
basic plan after the first disastrous night was for McNeil and Steinhauser to alternate 1 hour 
lectures. Since Steinhauser had taught math at the high school level for many years, he was 
comfortable with the assignment, and was well received by the students. 

What did the students have to do? 

Students were asked to read the relevant chapters before class. After class they had 
2-4 activities that were to be completed either individually or in teams. (Teams had been 
formed the previous semester and had worked out well.) Completed activities were 
submitted over the Web on WebCT. Appendix D contains 5 activities, one individual and 
four team activities. Activities 3.2 and 7.4, writing and responding to an RFP for an 
evaluation, were major activities for the course, and required the students to incorporate 
many concepts covered during the course. They were also very practical assignments, 
ones that they might refer to in their subsequent employment. 

How were the students evaluated? 

Appendix E contains the required activities and the percent assigned to each. Note 
that there was a combination of individual and team projects, a multiple choice exam, and a 
short answer exam. The short answer exam is in Appendix F. All exams and activities 
were submitted over the Web using WebCT. 

So, what did we learn? 

1 . ITV has a lot of technical problems. First, we started on time only once. Even 
though the two technicians started trying to connect the five sites an hour before class 
started, difficulties always occurred. Usually we started with less than the five sites, and often 
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would lose sites during the four hours. The next to last class actually started 1 1/2 hours late 
and we still were not connected to one site. The day of the last class, the entire lower third of 
New Mexico lost electricity three hours before class was to start. All radio reports were that 
power would be off for a long time. Since we needed power to transmit, we tried to come 
up with alternative plans. Since this was the last class, and it was close to the end of the 
summer, we decided to conduct the class via telephone conference. When we arrived at 
the Business Building, we were pleasantly surprised to discover that it was one of the few 
buildings on campus that had its own power source. So we were able to transmit (to all but 
one site). 

Even when sites were connected, problems arose. Transmission was over phone 
lines, so the image looked something like the TV phones from Afghanistan-delayed voice 
transmission and jerky movements. Consequently, we used a lot of overheads. The 
overheads didn’t move much and no lips were to be seen. This, though, was a different 
way of teaching. On the other hand, those overheads were easy to send via WebCT to 
each student. 

2. Meeting only 5 times requires a lot of planning, and a lot of material to be 
condensed into each session. Presenters need to be “up” for each session, and students 
lose a lot if they miss one session. Videos were made of each session, but distribution to 
each of the 5 sites and then to the 68 students took time. 

3. More program evaluation examples at the Community College level would 
probably have been instructive and appreciated. Most of our evaluation experience has 
been at the preschool and early elementary levels. We have conducted evaluations at 
other levels, including one at the Community College level. Even though most evaluators 
realize that tools and techniques can be easily transferred to various levels and content 
areas, students learning evaluation may not understand that, and probably appreciate more 
examples form their own field. 

4. The next course we teach in the distance education format will not be via ITV. 
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APPENDIX A 

Proposed Program of Study 

Spring 2002 

Community College Administration 
Community College Leadership 
Summer 2002 

Finance and Funding for Community Colleges 
Program Evaluation 
Fall 2002 

Higher Education Law 
Facilities and Capital Planning 
Spring 2003 
Statistics 
Action Research 
Summer 2003 

Dissertation Writing 

Student Services for Community Colleges 
Fall 2003 

Internship 

Doctoral Dissertation 
Spring 2004 

Internship 

Doctoral Dissertation 
Summer 2004 

Doctoral Dissertation 
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APPENDIX B 

Location of Distance Education Sites in New Mexico 



San Juan 



Albuquerque 



Clovis 



230 miles 



Hobbs 



New Mexico 
State University 
Las Cruces 



£ 



Carlsbad 
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APPENDIX C Schematic of Classroom 



Overhead 



Computer 



Video Equipment 



Instructor TV 



Remote TV 



Lectern 




Student 1 Student 2 



Student 3 Student 4 



Instructor Colleague 



Instructor TV 



Remote TV 
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APPENDIX D 

Examples of Activities that Students Turned In 



Activity 2.3 How can Stakeholders can assist in the evaluation? 



Product: How Stakeholders can assist in the evaluation. 



Time Process 

10 min 1 . As a team, make a list of the potential Stakeholders in your group 

evaluation. List at most 6 of these Stakeholders on the top row of 
below. 

30 min 2. As a team, review the Table of Contents for chapters 3-8 to get an idea of what the 39 

tasks entail. Indicate (with a Y for Yes) the tasks you think the various Stakeholders could 
assist with. Indicate with a N those tasks that the Stakeholder would not likely assist with. 



Total Time = 40 minutes 



Activity 3.2 Writing an RFP. 

Product: #1 Each team will develop an RFP. 

#2 Each student will develop an RFP. 

Time Process 

5 min 1. In teams, review the RFP in Exhibit 10.5. 

50 min 2. In teams, modify Exhibit 10.5 to meet the needs of theevaluation assigned to the team. 

(Only the INTRODUCTION and SCOPE sections need revisions. Make cosmetic changes 
only on the other sections.) 

3. Submit the RFP to the instructor within two weeks. Instructor will give feedback and 
assign the RFP to another team at your site. 

4. Assignment for next class. Each student will develop an RFP for one program, course, 
activity, or instructional technique. Other possibilities can be discussed with the 
instructor. This RFP will be submitted one week before the second class The RFP may be 
edited and then will be assigned to another student at your site. 

Total time = 55 minutes 
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Activity 1 .2 Specific differences between Program Director and Evaluator. 
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Activity 5.1 Concerns regarding instrument selection. 

Product: List of possible evaluation information for the team evaluation. 

In your teams brainstorm the possible kinds of information from Exhibit 5.2 that could be incorporated into 
the evaluation. Particularly discuss additional unobtrusive measures and try to include them. Then discuss 
for each measure: 

1. Financial viability 

2. Burden on respondents 

3. Disruption to program 

4. Potential gain from the information 

5. New insight as contrasted with degree of redundancy of that information 

Time Process 



10 min 1. Within teams, review the 5 evaluation questions 

30 min 2. Each team member takes responsibility for one of the above discussion areas and 

thinks about or talks with other members about the measure. 

25 min 3. Present to the class the team’s choice for evaluation instruments. 5 minutes on each 

evaluation question. 

Total time = 65 minutes 



Activity 7.4 Revised draft of the Team Evaluation Plan. 

Product: Revised draft of the Team Evaluation Plan. 

Time Process 

5 1 . In teams, discuss how the plan should be revised and expanded, based on the 

information already read and presented in class. 

45 2. In teams, write a revised draft of the Team Evaluation Plan. Realize that it will be 

modified in future classes. 

3. Submit the revised draft plan to the instructor within one week. Instructor will give 
feedback. 

4. Assignment for next class. Continue to revise and expand your own individual plan that 
has been assigned to you. 



Total time = 50 minutes 
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APPENDIX E 

Required Tasks and Percent of Grade 



Task #1 Required, but not graded. Each team will develop an RFP, modeling the one in Exhibit 
1 0.5. Due two weeks after the first class. 

Task #2 20% of grade. Each team will respond to an assigned RFP. The evaluation plan will be refined 
and expanded during the course. The plan will be turned in at least one week before each class after the 
June 7 class, and feedback will be provided before the class. The final version will be turned in one week 
after the last class. 

Task #3 5% of grade. Each student will develop an RFP, modeling the one in Exhibit 10.5. This 
individual RFP is due two weeks after the first class. 

Task #4 35% of grade. Each student will respond to another student’s RFP. The response will be 
developed during the course (using as a model the evaluation plan being developed by each team). The 
individual evaluation plan due the last class. 

Task # 5 20% of the grade. Activities will be assigned each class, and selected ones will be 
forwarded to the instructor and graded. 

Task # 6 20% of the grade. Final exam. Multiple choice and short answer. 
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APPENDIX F 

EMD 689 Final Summer 2002 1 hour 



1 . Relate the following quote to program evaluation: “Managers must have the discipline not to keep 
pulling up the flowers to see if their roots are healthy.” by R. Townsend 

2. Discuss the differences between Needs Assessment and Baseline in terms of Timing, Generality, and 
Involvement of Stakeholders. 

3. Why might the evaluation plan not emphasize evaluating training the first year of a five-year program? 

4. What are two disadvantages of open-ended questions? 

5. Many educators say that there is more within group variation than between group variation. Discuss this 
statement in terms of evaluating a course that involves 6 teachers. 

6. How could a high degree of mortality seriously affect the validity of an evaluation study? 

7. In what way is ethics an issue in selecting your sources of information? 

8. Discuss the negative aspects of an “inside hire.” 

9. Why do you think that “belief in cause and effect” is a requisite trait of an Evaluator? 

10. What are unobtrusive measures,” They are not used very often. Why do you think that is so? 
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